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Abstract 

Training new users of complex machines is often an expensive and time- 
consuming process. This is particularly true for special purpose systems, such as 
those frequently encountered in DOE applications. This paper presents a 
computer-based training system intended as a partial solution to this problem. 
The system extends the basic virtual reality (VR) training paradigm by adding a 
multimedia component which may be accessed during interaction with the 
virtual environment: The 3D model used to create the virtual reality is also used 
as the primary navigation tool through the associated multimedia. This method 
exploits the natural mapping between a virtual world and the real world that it 
represents to provide a more intuitive way for the student to interact with all 
forms of information about the system. 


1. Introduction 

DOE applications, such as Environmental Restoration and Waste Management (ER&WM) and nuclear weapons 
dismantlement will require complex, special-purpose robotic systems. Training the operators of such systems will 
be both expensive and time-consuming. This is due to limited access, the expense of duplicating the system for 
training, and the negative consequences of making a mistake while learning. This paper presents a computer- 
interactive training system that combines virtual reality (VR) with multimedia and artificial intelligence (AI) to 
provide an extended training environment for such applications. Virtual reality is used as the primary training 
tool, in much the same way that it has been used traditionally for flight simulators and other similar applications. 
A multimedia component, containing additional information about the application and equipment in the form of 
video, text, voice, and audio may also be accessed during interaction with this virtual environment. The 3D 
graphical model used to create the virtual reality is also used as the primary navigation tool through the associated 
multimedia. In addition, AI techniques are incorporated to aid in creating realistic simulations, in navigating the 
hypermedia and, eventually, in tutoring the student .The rest of this paper describes the prototype system. Section 
2 presents the application and the virtual training environment in more detail. Section 3 presents the multimedia 
component. Section 4 discusses the addition of intelligent agents for both training and media navigation. 

2. Virtual Reality for Training 

The initial application of this work will be in conjunction with the in-house Underground Storage Tank (UST) 
demonstration being developed at Sandia National Laboratories. This demonstration will provide a testbed for a 
number of technology development activities related to the retrieval of hazardous radioactive waste from 
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underground storage tanks. Because of the hazardous nature of these tanks, robotic systems will be used to 
retrieve and store the radioactive materials. This provides special challenges for robotic technology development: 
Tanks found in Hanford WA., for example, contain waste in the form of liquid, sludge, and hard salt cake. They 
also contain risers, pumps, cooling pipes, debris, and other obstacles. Obviously, the systems ultimately used for 
such work will be complex, consisting of many tools, sensors, and manipulators. Operating such systems will be a 
difficult task at best. 

The in-house UST demonstration testbed provides a mock-up of a UST environment. It consists of a variety of 
robots, sensors, and tools'. This entire robotic system is under computer control. At the highest control level, 
human operators use graphical programming techniques to provide commands to the system. Flat screen displays 
provide feedback concerning the status of the devices and sensors[Christensen 92]. 

An on-going development effort is underway to provide a virtual reality-based interface to this system. This 
interface provides a VR model of the UST testbed, including simulation capabilities for the manipulators and tools. 
The operator may move around in this virtual environment, using voice commands and hand gestures to program 
accurate graphical models of the actual robots. These task-level commands then automatically generate device- 
level programs to be down-loaded to the robots. The operator may also preview these operations for correctness 
via simulation and edit them interactively before sending them to the robotic devices. During operations, the 
simulation is driven by the positions of the robotic devices and the readings from the various sensors to allow the 
operator to monitor the state of the operations. The positions and states of the manipulators, tools, and sensors are 
accessed and used to upgrade the graphical models in the virtual environment. 

The virtual reality testbed for this work consists of a Fakespace, Inc. mechanically-tracked stereoscopic viewer 
(BOOM), a Polhemus FastTrack wrist tracker, a Dragon Systems voice recognition system, and a Silicon Graphics 
Inc. workstation with audio. The software platform is the Cim-Station robot modeling and simulation package 
from SILMA, Inc. The VR interface will be used both for training the operators of the system (simulation mode) 
and during actual operations (command and monitoring mode). The primary difference between these two modes 
is whether the actual robotic devices are in the loop or not. Figure 2 conveys the idea behind this work. The figure 
shows an operator in the Fakespace BOOM, the CimStation model of the UST testbed, and the actual testbed. 


3. The Multimedia Component 

The operator may access additional information about the devices from the virtual environment by putting the 
system into training mode. In this mode, a selected object provides an access point into a hypermedia system 
containing information in the form of video, text, and still photos. The hypermedia structure is a graph containing 
menu nodes for navigation and action nodes for accessing the media. In the case of the UST application, additional 
information is associated with each of the robots and with a 3D icon which represents the underground storage 
tank application itself. An object is selected by "touching" it with a 3D pointer driven by a joystick and pressing the 
joystick button. This device will soon be replaced by a hand-held, magnetic-based position tracker. Spoken 
commands will be used for making selections. Figure 3 shows the system with the various devices used for 
viewing the information. 

Information associated with each robot includes still photos, system diagrams, video of the actual device in 
operation, and the manuals for that device. Manuals are divided into reasonable chunks based upon the structure 
of the original text. Any figures associated with the text are also available. The UST icon invokes a text report on 
the Hanford site, photos of actual tanks, diagrams of remediation plans, and a DOE video describing the problem. 
Menus allow the user to navigate through the information for a particular device or icon. 
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Figure 1: VR for robot operator training. 



Figure 2: The VR-based multimedia training system. 
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Figure 3: A representative screen. 

Spoken word audio also provides guidance. A near-term addition will allow the user to speak preferences as well 
as to select them with the mouse. Figure 3 shows a representative screen for the multimedia component. 

4. Adding Intelligence 

Ultimately, the training system will embody intelligence in three forms: Intelligent behaviors during simulation, 
intelligent navigation tools for moving through the hypermedia component, and intelligent tutoring. Current 
emphasis is on the first two. The fundamental representation of an object is being extended to allow the invocation 
of intelligent behaviors for that object based upon the state of the simulation. For example, a pipe obtains the 
knowledge that it has been cut and behaves accordingly: it becomes two pieces, one of which falls to the ground. 
Likewise, waste contains the knowledge that it is radioactive and can communicate this state to the sensor object 
which can behave accordingly by creating the proper readings. Such intelligent behaviors add realism to the 
training simulator by allowing the user to run different scenarios. Emergency situations and equipment failures 
may also be invoked to train the operator to handle them. 

Intelligence will be added to the hypermedia component to aid in navigation. Initially, this intelligent navigation 
tool will utilize typed, weighted links and a set of simple rules to allow the system to provide a list of prioritized 
suggestions as to other, related information that the user might also be interested in accessing. Future work will 
enhance this navigation tool by integrating machine learning techniques. Learning algorithms will be applied to 
allow the system to learn from the behavior of previous users. The system will then use this knowledge to guide 
current users through the hypermedia information . 

Future work also includes the addition of an intelligent tutoring component that will interact with the student 
during training. In addition to information about the environment and tasks, the student will be able to access 
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interactive tutorials concerning the operations being t 
student's performance during a training session and 
critiques and advice concerning where additional train; 
may be found. 

5. Summary 

This paper has provided an overview of the intelligent, c 
National Laboratories for use in training the operators 
virtual reality with multimedia and artificial intelliger. 
complete teaching tool. The virtual environment provide 
robotic systems, such as those used for remediation oi 
model also provides the primary access points into rela 
form of video, text, still photos and sound. AI techniques 
the user navigate through the multimedia. Future won 
assist the student in learning about the system and tasks. 
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